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Abstract 

In order to judge the degree of confidence one should have In the results of 
an eKpetlment using eye movement records as daCai it is necesaary to have 
informatiori about the quality of the eye movaiaant data Itself* Suggestions 
are made fot ways of assessing and reporting this information. The paper 
deals with three areas* characteristics of ttia eye movement signal, 
algorithms used in reducing the daca,^ and accuracy of the eye position data- 
It is suggested that all studies involving eye movement data should report 
such information* Appendices include linear Interpolation algorittap for 
mapping from the eye movement signal to stimulus spacer and a way of 
obtaining an index of accuracy for each data point* 
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Evaluating and Reporting Data Quality in Eya Movement Research 

In recent years there has been an upsurge in the use of eye movement 
data in psychological research (Le^-Schoen ^ O'^Regan, 1979| Monty 4 
Senders, iy76i Rayner^ 1978)* There has also been considerable development 
In eye^movement monitoring technolo^, and at present "e are a number of 
technlquaa in use for collecting such data (for inita v j see Young S 
Sheenaj 1975), The process of, obtaining reliable and acG^rate eye movement 
racorcla is not an easy one, and there are many potential sources of error of 
various types* This makea it particularly Important that reports of eye 
movement research Include information which would allow Imowledgeable 
readers to assess the quality of the eya movement data obtained in the study 
and hence to be able to judge the d agree of confidence which they should 
place in the results of the study* So far, no general format has been 
proposed for reporting this kind of Information* In fact, while it Is 
ob^ous that information about the accuracy and reliability of the data 
should be presented, ^it is often not clear Just how to make such a report* 
The purpose of this paper is to suggest what Information inveatlgators 
should report about the quality of their eye movement data and to 
reeomrended ways of reporting "this information so that others can more 
effectively evaluate their research. 

It would be inappropriate to sat standards for what is and is not 
sufficiently good. eye movement data for reiearch purposes. The degree of 
reliability' and accuracy of the data which is needed for Investlgatihg 
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different questions varie t .mm studies only need Information 

about whather the eyes c .on or another, other studies need 

data on the durations o ^ Jtlll others need an accurate 

Indication of exactly V iuulus pattern the eyes were directed 

on a given fixation, 4 hase puts few constrainta on the quality 

of data, whereas the las great preclilon, which Is difficult to \ 

achieve* Thus, rather c mptlng to adopt standards, concerning what 

constitutes acceptable data, It will be more useful to make a list of items 
which might be reported in studies Involving eye movement data* This would 
provide a more or less standard basis for making comparisons among studies. 
An example of such a list is given In Table 1* Not all items will be 
appropriate for every studyi Rathers investigators should include those 
items whlqh would be necessary for evaluating the quality of* those aspects 
of the data that are used in their particular studies or research program. 

Insert Table 1 about here* 

The information which might be reported. falls Into three categorieai 
characteristics of the signal Itself, algorithms used for reducing the data, 
and accuracy of the data from which the results of the experiment are 
obtained* Each of these topics will be dlseussed below, with suggestions for 
the types of meaeures that would be appropriate. This discussion will be 
simplified by assTOing the monitoring of only a single dimension, the 
horizontal cofflponent of eye movemants. Carrespondlng information should be 
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repoifted for the vertical component If that Is Important In the study* 

Where the Inaluslon of both dimensions creates special problems In assessing 

or reporting the quality of data, this will be nc^ted- 

In order to standardise the datf quality measures. It will be assumed 

that the stimulus diiplay region Is dlylded Into a large number of small 

rectangular areas, all olE the same size,, by laying an imaginary grid over 

It. lach area will be referred to as an L-area, 'The width and height of 

each of these L^areas will be referred to as horizontal and vertical 

L-unltSg and these will be tiken as the units for measurement of the data ^ 

quality In reading research, for Instance, each L^^area holds a single 

latter* The L^areas moving horizontally across the page are referred to as 

letter positions, and the L^areas moving vertically, as lines. In most 

picture perception studies there are no such convenient elements In the 

stimulus display Itself ^ and the grld^produclng L^areas must be arbitrarily 

created. The use of L^areas and L^units permits the quality Indices to be 

reported in a more standardised fashion and thus permits easier 

interpretation of the indices and easier comparison among studies. 

. The first thing a report should include, theni would be actUFil width 

and height of the L^areas In mllllmeterSj thiia defining the horizontal and' 

I* 

vartieal L^unlts for the study. The width and height in degrees of visual 
angle from the position of the subject should also be reported for the part 
of the display nearest the eye. The viewing distance and the visual angle 
of the entire display should also be reported. Finally , the eKperlmenter 
should calculate thei amount of change in the eye movement monltpr (EIW) 

. ■. ■ ■ ■■(' , ... ^ , , 

V'J 
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aignal that typically results when subjects move their eyes a distance 



equlvalanc to one L-unlt, Thus, If the EMM provides an analogue signal 
which is digitized for storage^ this computation would Indicate the typical 
movement in these digitized valubs that occurs with a movement of the eyes 
of one L"unlt. If this varies considerably over different parts oJf the 
visual field, or for different subjects ^ some Indication of the range of 
this variation should also be reported. ' ' 

* i For future reference, the typical amount of change In the E^^^! signal 
resulting from moving the eyes one L-unit will be referred to as a Tinker ^ 
In honor of a prominent eye movement researcher* Thus, the Tinker Is the " 
unit of movement in EMM. data space equivalent to a movement of one L-unlt In 
the stimulus space. Of course, with Z-dlmenslonal eye tracking there will 
be both horizontal and vertical L^unlta and Tinkers* ' ^ 

In Bom^ systems, the E^M output la given directly In terms of the 
stimlus space, using Internal processing to map from the original eye 
position signal to the visual display. In this case, the units provided can 
be adopted as L^,unlts, and Tinker units would then be on the same scale* 

Characteristics of the Eye Movement Signal Itself 

There are five characteristics of the raw eye movement signal that 
should be investigated and reportedi the sampling rate, the delays maximum 
tracking rate, noise charactarlstics, and drift* y 



Safflpling rate * The time in milliseconds between taking successive 
samples of the eyea'^ positron should be reported. ^ ^ 
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Delay in ^he^ algnal * When Information about the eyes^ location becomes 
available for sampling, this Information Is necassarlly lagging behind the 
actual location of tha ayes* A good estimate of the delay In this signal Is 
Important for evaluating some types of research, particularly that Involving 
eye^movement^conclngent stimulus control. The amount of this delay is not 
always easy to estimate* However an estimate can be made on the basis of 
four facts about the eye movement recording apparatus and associated 
. equipment* First, how long does It take^the equipment to obtain the 
information needed to compute the eyes' location? For instance. If a TV 
monitor is being used to record eye movements, it may take 16 msec for the 
camera to complete a scan of the eye. In the case of llmbus reflection 
techniques, the information Is almost Immediately available* Other 
techniques typically lie between these extremes. Second, how much time 
transpires between the moment the information needed to compute the eye's 
location is available and the moment at which the eye position Information 
actually becomes available to be^ recorded or sampled by the computer* 
Delays may be induced at this stage by filters or signal processing 
requirements- Third, how long is it after the Information becomes available 
before the^ computer or other recording device actually has the sample* 
Delays at. this stage may result from slow sampling rates, from time required 
for digitizing an analogue signal, or from averaging over repeated samples 
for the purpose of reducing noise in the signal. Fourth, If the data are 

provided in one form (say, as values Indicating eye po^^ltion In the E^D^ 

■ ^ "ft " ^ ■ 

space) but tb be used Ewst be transformed to some other form (say, as values 

. \ , . n 

u . 

' ■ ' '' . 

o 
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indicating when the ayes are Gentered in the stimulus array), the time " 
required to make this transformation should also be Included in calculating 
the delay In the signal* ' .. ' ' * 

If the maximum tracking rate of the ElM equipment is too loWj this can 
also contribute to a delay In the signal during and Immediately following 
saccadic eye movements. ^Is problem will be dealt with in more detail. In 
the next section* 

In systems which give a stimulus position directly as output , these 
functions are handled internally and may not be available for test* In this 
casft, the manufacturer should provide precise indications of the delays ' 
involved* 

Information concerning delay in the signal is of importance for studies 
in wtftch stlisull are being manipulated In real time In response to 
characteristics of eye movements. When no such eya^movement^contingent 
stimulus control is taking place^ signal delay need not be reported. 

Maximum tracking rate of the eye movement equipment . During saccadic 
eye movements * the eyes reach velocities as great as 830^ per second 
(Alpern, 1971)* Peak velocities vary with the lengths of the saccades. If 
the signal pirpduced by the eye movement equipment is not .capable of changing 
fast enough to respond at the pealc velocity rates of the ^ movements 
typically observed in the task being studledj this c^ have several affects, 
A del^ in the signal will occur during saccadic mov^m^nta* The eye 

movemant velocltypattern obtaliied during 'saccades may be inaccurate, at 

iji' 

least *f or saccades above a certain length. The time duration of saccadic 
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movements may ba inflated, and as a result, the durations of flxationa may 
be underestimated, especially for flxstioni following longer saccades* 

A lower ffiaxlmus tracking rate can result from electronic filtering of 
the signal in an attempt to reduce noise, from equipment requiring 
mechanical movement In eye tracking * or other sources* 

Investigators should report the maximum tracking rate of the equipment 
they are using- This should be obtainable from the manufacturer or assessed 
by monitoring the movements of an artificial eye which can be accurately 
moved at different rates- 

Noise characteristi cs of the signal . There are two types of noise In 
the eye movement signal that should be reported- These will be referred to 

iocfll oolsfe and r^pe tl^t loiis pa t te rns » The first of these, local noise* 
conderns tri^ amount of variation in the EMI signal from one sample to the 
next when the eyes are Inffl fixation. It should be recognized, of course* 
that during a fixation there is some degree of movement of the eyes, and It 
would not be a reasonable goal to attempt to obtain a slgrtal that shows no 
change at all during a fixation. However* this movement tends to be very 
small with respect to the amount of noise fo^d in the signal of most EhOl 
equipment- ; . 

In order to estimate the amount of local noise present in the. signal, a 
series of fixations should be selected* and within these each successive 
data, value should be subtracted from the^ value 'obtained previously to yield 
a difference value* The absolute value of these differences' should then be 
obcalned- Information concerning the distribution of these values should be 
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reportad. This can be done by rtporting the median and the 90th parcentilea 
of this distributioni for instance* Dividing these indices by the value of 
a Tinker will transform them into a measure based on L-units and will 
Indicate the lavel of noise obtained relative to stimulus space units 
appropriate for the experimental situation. If the amount .of this 
variability changes from one part of the stimulus display to another (for 
Instances if greater variability is found as the eyes move into regions 
which yield the highest value), then distributions should be reported 
from both the low variability and high variability regions. , 

The^ experimenter should algo examine the raw data for repetitious 
patterns which may be present, but which do not show up in sample^ to^sampla 
differenced* For instances a 60 Hz nclse pattern resulting from changes in 
light intensity in the experimental roomj or line noise, should be noted, 
together with an indication of its extent. Again, the size of thli noise 
should be checked at both the low and high regions of the 11^ signal, and if 
there Is a difference, this should be mentioned. As before, the range of 
this noise can be converted to a more useful form by dividing It by the 
value of a Tinker. ' 

Drift . The final aspect of the eye aiovement signal Itself that needs 
to be assessed and reported is the drift. Often the EMM signal will change 
over time with no change in the stimulus conditions simply because of 
temperature changes or other factors that effect the electrical 
characteristics of the equipment. This should be assessed by establishing 
some type of standard stimulus situation which can be held constant for a 



■ ^ . ' ' V ^ ^ ^ ' Dsta Quality 

' ' . " . 10 

period' of time* This may involve Che utf^ of a stationary artificial eye, 
for instance.' The equipment *ffiould th^n be adjusted to provide an output 
signal In the low range of the Eb^ signal, and It- should be sampled 
ragul^ly, say every 15 seeondi, over a period of time equal to that 
typleally required ;for a subject to complete the experimental task being 

_ _ r . . — - . ' // . . \ 

Studied. This same test should- be repeated with the equipment adjusted to 
yield lan output at the- high end of the'E^M signal range* The timing of this 
test should^ be similar to the typical use of 'the equipment for data 
collection* -That Is, If data are typlcaily collected immediately after the 
eye movement monitoring equipment is turned on^ the test should be made the 
same wiy; If the equipment Is typically allowed |© warm up for a period of 
time, the .test should be done after similar wafm^up* Data from "this test 
should be included in the description of Emi signal characteris tics* 

, Summary * ^ ^The report of suggested information concerning aampling rate, 
delay, msKlmum^ tracking rate, noia^ and drift in the signal will help 
readers understand some of/ the* probieras encountere4 by-th6 experimenter in 
making- decisions about when fixations began and ended, where the eyes were 
directed, etc. Some of t-he problems Involved are discussed further by 
McCohkie, Zola^ Wolv^rton, and Burns C1978), . ^ ^ ' ^ 

Algorithms^ Used In Reducing the Data" ^ \ ^ * 

Eye ffloyement research often requires four algdrithms that convert the 
r^w data to data showing a serles-of fixations at particular stimulus 
locations. Some studies do^not need all four types of iniormatlon and hence 
do not require algorithms of all four types, Thi algorithms are for ; 
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(a) Identifylhg the BeginnTng of a saaoade Xpr end of a fixation)} 

(b) Identifying the end of a saccade (or beginning of a fiKation)} 

(c) identifying inhere in the stlmulua display the eyes i^ere direcCed during 
that fixation^ or identifying the direction and extent of a sacoadei and 
fd) Identifying* disturbances in the eye movement data that auggedt that the 
data should not be used (for instance, blinks^ squints, ,or other 
irregularities). The nature of the algorithm which must be used to 
accomplish each of these depends greatly on ,the characteristics of the 
algnal itself, particularly the level of noise, and on the nature, of thi - 
calibration task used and the type of Ijiformatl^ which it provides for use 
±n transforming the data from EJfll signal spaci to stimulus display space* 

The algorithms used for these purposes ^ insofar as they are applicable 
to the study being^ reported, should ^e des^^ey^,, or ref ejfenae should be 
nade to some source where they are publicly avallabre* V ^ . 

Examples of algorithms for taking a linear iijterj^olatlon approach to 
map from EMi signals to stimulus locations (that Is, to indl^cate where in « 
the stimulus the eyes were directed at any glve*n moment) are given in 
Appendices A and Appendix A prraents a common simple algorithm for use 
in one^dimensional eye trackingi, an^ Appendix B^presents an algorithm for 
^use when both horizontal and vertical coiAponents of eye w^ements are being 
monitored. V ■ - ^ 

The use of such algorltMs as. these require tha^ ^the subjeet'^be eniaged 
in some sort of calibration task which yields a set of EMM signal values 
that correspond to a set' of known stimulus locations* The algorithm for 
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mapping EMl*slgnal^ to stimulus locations (which will be referred to here as 
* oapping algori thm ) simply provides a means of Interpolating between these 
known points to assign stimulus locations to other EI^ signal values - The 
calibration task which will be used Here as an pxample Is to have the 
subject look directly at %ach of a serie^of points, and while looking at 
eachj to press a button. This cauS-es the computer to pample the signal 
value corresponding to each stimulus location arid to store the value In a 
table J referred to as the calibration table . Following the calibration task 
this table oi numbers is used by the mapping algorithm* Other -tasks can be 
usedj of courjse, and this may change the nature>pfjrte algorithm used for 
mapping (for instance^ see O^Regan, 1978). The nature of the calibration 
task and<of the resulting dallbr^tion table should' be reported. 

It should be pointed out that linear lnterpo3^atlon approaches of the 
type described in these Appendices make two strong assumptions. First, they 
.assume complete^epeatabllity of the ItM values obtained during the 
calibration task. Second,^ they assume thatj within each stimulus region 
bounded by aiijadent points used in the calibration 'task, the distances 
between real fixation locations and the differences between the EMM values 
corresponding to each of these locations are linearly related. T^ the 
extent that, these assumptions are violated, the accuracy of the data, in 

i ■ ~ ' ' - 

terms of where the eyes are being directed In the stimulus or In terms of 
the absolute lengths of saccades, Ls brought into question^ • . ' 

Some suggestions can be made or improvement of the accuracy of this , 
aspect of the data. First, great care should be made in obtainilig 
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repeatable^lMM values for each flKatlpn target location during the 
calibration task. Subjects must ,often be trained to exercise care In this 
aspect of an experiment. Ofla way of doing this is to consistently pfbvlde 
them with feedback concerning the degree of repeatability they are showing.. 
In this way, subjects can. be engaged In a sort of game. of Improving their 
own. performance on this task. Another technique that can be used Is to have 
phe subject^ fixate each target location more than once during the 
calibration period* Then If the EMI values obtained from the same fixation 
target location are not sufficiently similar, the sijbject can be required to 
fixate that location additional times until successive values are close 
^ enough to meet the criterion set- In this wayj spurious values are 

rejected^ and greater consistency is obtalned.^ If this technique is. use^ in 

^ ^- ^ ^ , - - ' ■ ' I ' - 

an experiment j the investigator., should report the criterion used for ,7 

accepting Em values during calibration* . ^ / 

^ : / . ■ - . . K 

It has been our experlende ..that one source of spurious values during 
"calibration arises from .subjects' tendency to move their eyes away from the 
fixation point too quickly.^ If the task is to look directly at a point and 
press a button^ subjects will often Initiate a saccade before the button is 
pressed p This tendency can be greatly reduced by having oiily a single 
fixation target available, at any one time. After each 1^ sample Is taken, 
the target Is then moved to a new location.' In addition, a tendency for the 
subjects to "'anticipate the move of the target , again making saccades prior 
to pressing /the button, cbn be reduced by leaving the target in Its present 
location for about 500 msec after the button is pressed, and only' then 
moving It to Its' next locatiom " / ' 
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Given that reliable ElM values arfe tecorded during the calibration 
taak^ there la still the problem of dealing v#lth ndnllnearlty in the E^M 
.signals The preaenee of nonllnaarity , when using a llneat Interpolation^ 
mapplni approach Hk^i those presented in Appendices A and Bg has the effeet 
of produelng error in the aecuraey of mappfng f torn E!W values to ^stlmulue 
looatioha in thos^ regions between the fixation' target locations used during 
calibration, M approach to assessing the amount of this error in a given 
experimental situation will be described in the na^kt section, "nie amount of 
error can be reduced, of course, by using more fixation target locations' 

* during calibration, and by concentrating the density of these locations in 
the regions of greatest nonllnearlty. , 

V Th# pthar a to dealing with nonllnearlty Is to abandon the use 

of linear interpolation techniques. It is hoped that those researchers 
using cutvllinear Interpolation techtilques for mapping from EI#I values to 
Stimulus locations will be encouraged to aescribe tli^se techniques In print* 
O'Regan'^s (1978) smooth pursuit approach avoids all interpolationi given 

^ that movamentjiri only a single dimension Is being recorded. Having. , 
alternative approaches available will provide new Investfgaf ors with a 
selecti6n from whlqh to choose tWe most^ appropriate for their purposes, 

given the constraints of their research (accuracy pequirflmenta, time or 

\ ' - - ' . \ ' ' - . 

cbmputer space limitations, etc*)* ' ^ 
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Accuracy of the Eye Posltioii Data * ^ ^ , 

, There la of ten some* aonfuslon about the meaning of accuracy when 
speaking about eye movement data* O^Ragan (personal eoffijminlcation) has 
suggested dlstirigui*shlng between relative accuracy and absolute accuracy. 
Relative accuracy refers ±0 the reiolution or ienflif:ivity of the Elm 
equipMent; that is, how amall a dlsplaeemant of eye position can be reliably 
detectadi Absolute accuracy refers to the ability of ithn system to 
determine the orlentifction of the ayes with respect to locations in the 
visual field* *v IWl equipment can have very high ^relatlH;^, accuracy 1 yet be 
poor in absolute accuracy, for a number of reasons* It is i^nportant^ that 
comments on ^accuracy indicate which type is being discussed* In the present 
context, the term accuracy will refer strictly to absolute accuracy* 

Souf'ces of Inaccuracy in eye ppsition data can be grouped Into three 
categories* First is error which reduces short-term repeatability of the * 
eye movement signal* Thls^lncludes noise in the 1^ slgrikl, inability of 
'subjects to reposition their eyes accurately, etc* and htnce Iliads to 
variation in eye position Values -when the person is asked to succesisively 
fiK^ate the same point. Second is. error introduaed in mapping from EIM 
•values to stimulus position, ^is primarily results from usiAg an algorithm 
bhat is. inadequate to "deal with iKonlintfar^ty in the calibration matrix* 
Third is error ^hlch develops over time during ^the ^sperirtental task, and. 
might be called longer^teM repeatability. Due to head movement,, electronic 
drift, or other factors, calibration values obtained prior to the task may 
differ from those taken following the ^ask* * 



Data Quality 
^ 16 

Degree of short'^term repeatability * The Ideal eye movement monitoring 

_^ ' . ------- ^ J 

situation would be one in whleh the EBW signal returned to exactly the aame 
value every time a* subject was afked to look directly at the satfe location. 
That le an Ideal which is not reached for a number of reasons* Some of the 
reasons were dealt with in a prior seetloni E^W signal noise and drift* 
However, other reasons coul^ include varying lighting conditlona in the 
eKperimental room, head movement , pupil size ch^pgea (which may result from 
changes in the amount of light emanating from a CRT display or from 
pupillary responses to processing actlvltlas), changes In eyelid position 
(especially when eyelashes Intrude into^ t^e sensed region , amount of f l^ld 
on the eye's surface (which may vary with the time^plapaed since the last 
blink,, or with whether or not an air conditioner in the^ room is on at a 
given moment), various types of problems in the dynamic operation of ..the EWW 

^ijtself, or lack' of consistency in the- position of ^the' subject's eyes iwhen 
asked to look directly at the'samiB location* Thus , an indication of the 
amount of variance in E^M signal values obtained when the subject looks 
f^epeatBdly at the same point gives a general simmary of , the quality- of the 

entire eye movement monitoring altuatlon* . 
For a one^^dimenslonal eye^tracking situation I this can be donfe by 

conducting a task like the calibration task described earlier In which three 

^ ' _ ^ ■ . , - r . ■ . _ ' f * ■ 

to five, fi3catton points are displayed at equal distances apart, with the ^ 

eKtreme points^ being at the ou^tslde edge of the stimulus region within which? 

eye movament- monitoring occurs in the experimental situation* The subject 

is then, asked to look directly at each point In succession. If a cathode^ 
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ray tube (CRT) Is used as the display device, 'a^target (say a dot with a box 
around It) can ba made to appear aucaaaslvely at each of these points In 
aucceaalon or In soma random prder* aubjact is asked to look dlractly 

at each dot In each locatldu and press a button. The El#f signal value 
should be obtained corresponding^ to the time of each button pWss (given' 
^ that the eyes are tn a fixation). If the signal Is quite noisy, an average 
over several EMH values following the bufton press should be obtained to 
tndlcate the Elm signal obtained when the ayes are directed to that point* 
Thl^ Is done repeatedly until the subject has looked at ,eaehi point, say, 10 
times* Each successive E^W value can be subtracted from the previously 
o^tallfted value corresponding to that point to yield a difference score # The 
standard deviation of the distribution of these error scores can be 
obtained* This standard daviatlon than becomes pn Indication of the degree 
of short tietm repeatability of the data. Furthermore, If the standard 
deviation Is then divided by th& Tlnket value ^ the lndex\ of repeatability is 
transformed: to an L--unlt scale* ' \ 

J^^^^- three added compleKitles, First, different subjects may "show 
different degrees 'of variability In such a measure of repeatability, since 
the measure depends on thelri ability to adjust their eyes to the same 
position. when looking at. the same location, and with some equipment on their . 
abflity to keep their head motionless. Thus, It may be best to have arrange 
of r standard deviations obtained from several subjects* Second, the amount 
of variability may be different at different regions in the visual f laid. 
Often the EMM values obtalhed when a subject Is looking to the outer areas 
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of the iegion within which the eyaa are being monitored tend to be less 
atable than when looking at the more eentral areas/ nius^ some indieatlon 
of the range of standard deviations obtained from, different areas in the 
visual field should be Indicated If there is dubstantlal variability. aUso, 
the experimenter should. report any patterns observable (for instanoe, a 
tendency to have less repeatability In particular regions)* Third, the task 
as described may iiot tap some sources of variability present In a given EWi 
system. For Instance, If pupil slge changes affect the indication of eye 
positions then the eye position recorded may depend partially on the amount 
'of light coming from a CRT display being viewed by the subject* This could 
occur In ;a reading eKperlment If one page of text were shorter than another,| 
thus reducing the total Illumination coming from the CRT* *nie .effect of 
this variable could be assessed by having the subject look repeatedly at the 
same set of points, as Indicated earlier, but also adding and deleting 
extraneous material on the CRT to change the tbtal Illumination available ^at 
different times. The effects of some other possible variables can be 
assessed in thl* same way. ' ^ 

^ JEf the ayes are being monitored over a two-dimensional area, standard 
deviations should be calculated for both horlEontal and vertical^ mtasurss ^f 
eye position separately. Furthermore, this process should be repeated with 
the row of fixation points occurring at three to five dlffereht vertical 
locations, in order to test repeatability over the entire area within which 
eye movement monitoring la taking place* ^ 
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While this measure of ra^atablllty gives some indication of the total 

■ , "'\^ ■ ' . . / 

iystem performance^ It la of particular interest in dealing with data when 
the calibration procecjure used the actual research la similar to that 

described above. ^InforMtion 4>out repeatability provides ope indication of 

" ^^^^ ' \^ 

the degree of faith one should have in the accuracy of the mapped data 
values, and whether one can have more faith In the accuracy of data coming 
from certain regions of the display area than others* Other indicationa 
will be described later, ^ ' 

A second way of providing repeatability data is to collect the data 
.during the experiment itself. In this approach, subjects are asked to look 
at each fixation target twice or mbre In the qalibratlon prior to the 
eKperimental task, and then twlbe or more limedlately following th©^ task, 
thus yielding at least two pairs of values for each fixation target 
location* Difference scores are the.i coinputed by subtracting the first of • 
each pair of successive values from the second. The standard deviation of 
the 'distribution of these differences can then be reported, divided by the 
Tinker value to convert to the tf-unlt *scalej as described earlier^ 

Accuracy of the mapping function . The secdhd of thass sources of. 
Inaccuracy, which resul^ts from the mapping algorithm, should also be ' 
assessed and reported. This can be done In the following manner. Firsti a 
calibration task Is-^used in which the subject is asked, to look directly at a 
series of points and press a button, with the^ computer aampllng the EM 
value cofresponding to each stimulus location. This series of points should 
Include those locations used In calibration In the normal experimental task^ 
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plus poin£s halfway between each of theses which we will refer to as mid-" 
polnte« SeQond^ the mapping algorithm should ^then be used to assign 
stimulus locations to each of the mld^polntSj using only the calibratlpn 
data corresponding to those points normally used in calibration in the 
eKperimant* Thirds the location of each of these assigned stimulus 
locations should be subtracted from the actual locations of the - 
corresponding midpoints to produce error scores. The distribiitiQn of these 
error scores then indicates the degree of cbmblned error from the lack of 
short-term repeatability plus inaccuracy In the mapping. This can also be 
accoi^anied by some indication of the degree and nature of the nonlinearity 
typically foiind In the calibration tablep so the reader can have some | 
impression of the types of distortions with which the mapping algorithm was 
faced* ' . ~' ' ^ ^ 

Degree of longer-term repeatability . TJie third souce of inaccuracy 

has to do .with those factors that can change over the period that data are 

^ ^ ^ , ^ .... 

being collected during an e^erlment^ Including head movement,^ elect ronlc 
drlfti etc. The degree of inaccuracy from these sourcea can be observed by ^ 
ehgaglnli subjects In the calibration task both before and aft^r data 
Collection, and coiqiaring the calibration tables obtained by subtracting' 
corresponding values from the pairs, of tables. This yields a distribution 
of error scores reflecting both short *tarm and longer '^term repeatability. 
The mean and standard ^evl^tion of, this distribution should be reported. 

Often this third source of error is the greatest contributor to total 
Inaccuracy In the .data,j^^resultlng primarily the effects of head 
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movement. If It can be 4emon§trated that the degree of Inaccuracy •^esul 

from the first two sourcea is relatively small, then it is possible to 

obtain an Indexlfor madh data value which Indieates its degree of Inaccift^acy 

due to this third soureep and its level of accuracy In. general. Such an 
* * .5 ' . ' . ^ ^ 

index can he particularly useful In reporting the level of accuracy of data 

for a**partl^ular experiment, or in selecting only those data which show an 

acceptable level of/ accuracy required for the exp'erlment being conducted* 

In order to obtalK this measure, it is first necessary to perf drm a 

calibration task both before and after the experimental task. In this way, 

two sets of EMM values are obtained which correspond to particular atimiilus 

locations, one. prior to the eKperlmentaL task and one following it. 

Second, th& assumption Is made that during the task used in the 
experlnient, the E^W signal values associated with any given stimulus point . 
^ange betwen t4ioge which wduld be assigned by the calibration table values 
obtained %e^fore the task, and those which would; be assigned by the ^ 
calibration table values obtained after the task. 

t^lle this assumptflon is undoubtedly violated at times (for instance) 
the subject's head mAy move in one direction and then return before the end 
of the task, or drift in the signal may proceed In one direction and then 
ret4irn) , we do'npt Kave direct evidence of fiuch events,' and they will be 
assume^d to occut sufficiently seldom to permit their being ignored. 

Given this ass^ption, an accuracy indicator IndeK can be obtained for 
any given data value'. To do this, It Is first recognized that three 

* _ - ■ ■ 

dlf|erent sets of calibration values can be used to map a given data value 
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onto a' stlnului locatloni Che values obtained prior to the experimental 
task, those obtained following It^ and an average of these two* Using 
these, an E^0! data value can be asalgned three^ different stimulus locations 
through some mapping algorithm su^h as those presented in Appendices A and 
ft* since we do not know which of these locations is the most accurate (that 
isj w|iich corresponds most closely to the *'true" position of the^ayes at 
that time) , the location obtained by using the averaged calibration data 
will be taken to indicate the best guess. Howeveri taking the absolute 
value of the difference 'between the other two locations indicates the range 

. of ^^pcertaihty of the location corresponding to this data point. Since the 
location obtained from the averaged calibration data Is halfway between the 
other two locations, a simple indicator q£ data accuracy is computed by 
dividing the range of uncertainty by. two* ' It should be noted that this same 
value would be obtained by taking the abiolute value of the difference 
befcfeen the location assigned by the averaged calibration data and plther of 
the other locations* Thui,' It is not nieessary to compute all these 

4 locations. This index, which will be referred to as, the IndeK of Accuracy 
(fA)i indicates * that the stimulus .^location assigned to that' EJW idata' value . 
by .using the averaged calibration values may be off In either . the positive 
or negative direction by ''an amount Indicated by the index* Thus, If the 
three locations which ape assigned to an E!M data value of 2037 are 45.7, 
47.2, and 48,7, we would take 47.2 to be the location "of the eye (that is, 
the eye is oriented to a location 2/10 of th# way acfoss the 47th L^area. 

^However, we would also indicate that this location may be off by as 'touch as 
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1*5 L-u^lts to left or tight. If the experiment require/ accuracy of 1 L- 
unlt or less^ this data pdlnt would be excluded as not having the needed 

_ W , V ' * . . * 

level'of accuracy. ^ 

Of course^ tha same, procedure can be followed for calculating the lA on 
,the vertical dimension for any data value* In two-dimensional eye trackings 
a data point may be. rejected because of failure to obtain sufficient 
accuracy on (either of the two dlmensloni. * .. : 

A fornaj^la Is preseated in, Appendix C for directly Calculating the lA 
for any data value when dealing with linear Interpdlatlon with ' 
unldimenalonal eye trackings without having to calculate multiple stlmuliis 
locations for each data value. With mdra cooplex mapping functions, It will 
of ten be hflcasaary to calculate the 1^ In the manner described above. ' 

it should be noted that in packaged systems whloh do not make 
calibration Information available to the researcher, but simj^y use It 
internally to map eye position daftf onto the stimulus spacei It is not 
passible to obtain such an IndeK. It Is particularly Important that 
Accuracy •be •capefiilly assessed with these systems , using' techniques similar/ 
to^those deiscrlbed earlier, since inaccuracies are often not readily A 
apparent In data normally collected for eKperiments. / 

' , ' ■ • . ■ " ' : ■ ' ' ■ , ■ ^ J 

f ConcluBlon ^ 
Thk present paper; attempts to encourage standards In the reporting of 
psychological resiiareh involving eya movement data* It argues that it Is 
niat appropriate to adopt standards concfernlng what is acceptable datai since 
that varies with the nature of the questions ibelng studied. However, It Is 
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approprikte to list the irifprmation whleh ought to he reported by 
reaaarcKers so that others- can judge the adequacy, of their data. Thus, this 
is an argument for standards in the reporting of dataj rather than standards 
concarnlng the data itself. . . 

' If inves^tigatprs^ engaged in eye movement research will use these 
suggeitions to make a rather complete report of the quality of eye movement , 
data obtained in their research^ there should be. several'^^ desirable results* 
First, other investigators ^11 have a basis for Judging the adequaey .of the 
data reported in an eKperlmenti given the nature of the questions being 
investigated* feeond^ Individual investigators will begin to have standards 
In. the published literature against which they can judge the adequacy of 
their own data* Thirds this is likely to put pressure on both investigators 
and equipment manafabturers to Increase tlie data quality of their eye 
movement monitoring .equl^ent. 

In addition, it is our hope that these suggestions will provide the 
beginnings pf ^vocabulary for discussing the quality of data being obtained 
In this burgeoning -research area* , \^ . 
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. ' ^ Appendix A ' ,. 

Linear Interpolatlbn Approaah to Mapping an Value 
onto a Stimulus Location in One Dlffiension , 

It la assuined that a calibration task has been performed which provides 
E!#f values which correspond to certain known points on a single dimension In 
the i^lmulus arra]^^ The location of these stimulus points Is given in a 

vector L^s The location of eafch of these paints Is given on a scale of L- 

' th 
units J where the left boundary^ of the ^ L**area has a location value of 

Thus a point at th^ center of the 5th L*area from the left of the display 

has a location of 5 #5. This type of scale allows for easier computation 

with the data later, since taking the Integer value of any location^ without 

rounding, indicates the L-area w4=thin which that point lies. . \j 

The values corresponding to the locations in L^ are contained in 

vector ^* Thus J contalna the value corresponding to stimulus * 

location L^ * . 

In mappings or translating, a given EMM data- value _D to a stimulus 

position £j it Is first neceasaEy to locate the last value in which Is 

equal to or smaller than D, This value will be labelled E , and 

— ' "=in 

D £ D < E., . This also indicates that L 5 S < L . The value of S can 
be obtained by using the following comion linear-Interpolation formula: 
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At times an WW value ma^^ba obtained which falls outside the bounds 
given by E^^^ and E , wheifa n Indleates the number of entries In and 

S ^ El> the Interpolation can be suceeaafully carried out with m « 1^ 
and when D > E the interpolation ean be carried out with m ^ n - 1, Of 
course, the accuracy of the resulting £ locatloni becomes more suspect the 
farther they fall outside the region within which calibration data were 
obtained* 
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Appendix B , 
Linear Interpolation Approaeh to Mapping an Ehtt! VAlfVe Pair 
onto a Stimulus Location In ^o Dlmahsions ] 

It la asamned that a calibration task has bten performad which yields a 
set of E^fl^ values which correspond to certain known prints In the itlmulus 
array. For slmpllcltyj , we will assiMe. that these looatlons form a grid over 
the stimulus, being arranged In re^lar columns and rows. As in Appendix A| 
the loeatloni of thepe columns and rows are given In L^unlts* A column of 
stimulus points at the leftmost boundary of the 1^^" column of L^areas Is 

given a horizontal location of 1^1 a row of stimulus points at the bottom 

h / ^ i • 

boundary of the t : row of L^areaa (count^^g from the bottom) la given a 

vertical location of 1^. Thus a jfoint at the center of the bottom left 

area has a horizontal location of 1.5 arid a vertical location of 1.5. 

•. - / ^ 

The horizontal and vertical locations of each of the points fpt which 
MM values arex^nown are assumed to be contained in two vectors * LV which' 
contains the verfrtcal location of each of these points, and LH which 
contains the horizontal location^ LV contains r values i the number of rows 
on which calibration values were obtained, LH contains £ values, the number 
of columns in the calibration task. 

The horizontal values associated with each of these stimulua 
locations Is assimed to be contained In a matrix, having, rows and ^ 
coluranQ. A second matrix of the same slEe, EV, contains the vertical- 

values associated with each stimulus location# Thus, the horizontal and 

th th 
vertical EMM values Qorrespondlng to the J calibration point In the 1^ 
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row are contained In EH . . and EV. Theie values can be used to blot each 

Insert Fipire 1 about here* 

calibration point in Efffl value space^ as shown in Figure 1. Here the seale 
on' the X-axis is the horlgontal EMM values, and the scale on the ^'axis is 
the vertical EMH values* It can be seen that while the original calibration 
stloiulus locations may have been arranged in a rectangular grid pattern, the 
corresponding locations in the Emi space may not be. The fictitious data 
shwn in Figure 1 are highly nonlinear. ^ * 

Figure 1 alsd shows a particular 1^ data pblnt D^, having vertical 
position D and horizontal position D. for which" a corresponding stimulus 

^ ^ ^ . .. ... ^ _Q 

location S^^ having vertical position and horizontal is desired. An. 
algorithm for mapping £ onto using a linear Interpolation app . . h, will 
now be describe<f. \ ■ 

First It is necessa^ to deterMne which region of the EBW value space 
shown in Fi^re 1 contains the location D. This region Is shaded In the 
figure and shown In enlarged form In Figure 2. This region can be found by 

Insert Fi^re 2 about here. ^ 

using a steppliig,.algorithm. In order to use this algorithm, it la necessa^ 
to calculata the slope and Intercept for each line segment shown in Figure 1 
connecting two successive^ calibration points In either the horizontal or 
vertical direction,^ 
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The st^Dplng process begins at the lower left cotner of the pattern, at 
and the first step is In the horlgontal direction to A ^. At that 

point, we ask whether the data point £ lies above, on, or below the line 

_■' i 

passing through ^ and A, If it lies above, 1^ should be incremented 

1|1'1|4 

for the next stepi if It lies on or below, J^should be Incrementad* In this 
case, it lies above, and the next step goes to Ag Again, the data point 
D is compared with the line Just traversed (A^ ^2^^* this time to 
determine whether D lies to the rights on, or to the left of that line. If 
to the left, !_ is decremented (if 'possible) | If on or to the right, it Is 
incremented* Thus, on each step, the data value is compared with a line 
passing through the last arc traversed, and the next step is in the 
direction of the data point from the line i that is, i_iOt is^ either 
incremented' or decremented appropriately* Where movement in that direction 
is impossible (as when lies below the bottom line of the cfilibratlon 
pattern), movement continues in the same direction as the last step* At 
corners, movement goes In the only direction possible- Moving back to the 
immediately prior step is not permitted* 

k 

At the same time, a history Is kept of the points visited In this 
stepping* When the algorithm results in a return to a point previously 
visited, this point and thj^ prior three points visited will be found to 
specif y the region within which liesi or which should be used^ for mapping 
when.D^ lies outside the calibration area* The stimulus ^location of the 
calibration point at the lower .left corner of this area will be labelled 
LV . LH for the remainder of this section* It has a corresponding EM 
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spaca locacion of EH^ EV^ ^* The four lines bounding this area, -the four 
Dolnts defining those lines, and the four corresponding points in the 
calibration stimulus array, are used to map onto a stimulus location S^. 

In actual practice, while it is necessary to use thm stepping algorithm 
just described to find the Ideation of the first EMM data value, each 
suceesslve value can typically be properly located by testing whether this 
new value has crossed the boundary of the region containing the last value » 
In the direction It has moved fromr the prior value. ' - . 

,^ Once the region within which is located has been found, mapping to a 
stlmilus location §^ proceeds by linear interpolation as shown in Figure 2 
and described In the steps described below* Fo^ ease of communication each 
point has been given a single letter designation. It is assumed that the 
slope and' Intercept of the lines bounding the region, TO , WY , XZ , and 
have already been calculated and stored in a table. * j ^ 

i; Find P, the point #here and XZ intersect. If and ^ are 
essentially parallel, flag instead. 

2. Find G, the point wh6re VK and Y2 intersect. If m and YZ^ are 
essentially parallel, flag £ Instead . 

3. Find the slope and intar^ept of line FD. If Is flagged , take 
(ha average of the slopes and Intercepts of ^ and XZ Instead. 

* 

4. Find M| the point where FD intersects YZ. 

5. Find N, the point where TO^ intersects TO . 

S. Find the distance f rom !l to D^, as a proportion of tha^ total 
distance from N tjo 
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7, Vertical position of £ is given byi 

where S is the vertical positloa of the fixation In the stimulus 
apace measurad in L^unlts from the bottom row of L-areas in the 
stimulus space*. ' 

LV s the vertleal location In L^unlta, of the fixation targets 
corresponding to polntp W and in Figure 2* 

LV_^. ^ the vertical location, in L-unltSs of the fixation targets 
corraspoadlng to points and in Figure 2* ^ 

8, Find slope and intercept of line CD. If G is flagged, take the 
average of the slopes and inter cap ta of TO^ and YE^ instead* 

9* Find H, the point where GD intersects with ' 
10, Find K, the point where intersacta with 



II, Find F. , the distance of H to D, as a proportion of the total' 



^ distance from to K. 
12. Horizontal position of £ is given byi 
^ LH + P, (LH - LH ) 
where S, Is the horizontal position of the fixation in the 
stimului space, oaasure In L-unlts from the lef t'^ost .colusm of* 
L-araas in tha stlmlus space. 

LH » the horizontal location, in L-unlts, of the fixation 

targets correspohdlng to points W and Y in Pigura 2. 

LH « the horizontal location, in L-unlts, of the fixation 
n+1 

targets corresponding to points X and Z in Figura 2, 
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AppendlK C ' 
^ Calculating the Index of Accuracy (lA) for a EMH V^lue 

It is asst^ed that a translatioii matrix has been obtained both before 
and following the task by getting E^W values resulting from looking at 
certain .stimulus locations* For the present, we will deal with eye movement 
monitoring on a single dimension, assumed to be horizontal. Thusv the 
calibration matrlK in thlp case will be in the form of a vector. 

Flratj there Is a vector L^, containing values Indicating the location 
of the points In the stimulus for which calibration Information Is obtained. 

will contain as many values as there are points on this' dimension for 
which corresponding E^W values were obtained. The values in L will be in 
L*unlt8. 

Nextj vectors of EMM values corresponding to each of these stimulus 
locations are defined. Vector EA ^ with values Ek^ . * . EA^ • . . EA^ ^ 
i^ere there are c_ stimulus locations used in calibration, Is the vector of 
calibration values obtained before the eKperlmental task. Vector EC , with a 
similar number of cells, contains the calibration InforMtloti obtained after 
thm task. A vector EB^ Is obtained by averaging the corresponding values of 
EA and EC , and another vector Is obtained by subtracting each value of EA 
from the corresponding value %n EC . Thus ED Is a difference vector* 

Finally , ^ there Is a EMI value, D^, for which we wish to obtain an Index 
of Ac curacy (lA). . v 

The first step Is to find the last Valut of which is smaller than D. 
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Next the following formulae are used to calculatfe lA for S^: 

where ^ is the number of values eorrefipondlng to a single L*unlt In this 
region of .the' stlmului pattern, or the local Tinker value. 

W - R(ED, ' . 

lA - (|D^ + W)/2£ . , . . 

The^ result Indleates that .the translated value corresponding to data , 
value D Is accurate to plus or olnus lA L^unltSj If short-term repeatability 
is high and the mapping algorltta yitlds minimal error* 

If ^ two-dimensional eye tracking Is being carried out, a similar 
technique may be employed to yield lA values for both hfirliontal and 
vertical components. In this case^ however^ It Is necessary to think of the 
EMM data space as being divided Into quadrangles 5 with four corners defined 
by data values corresponding to the four points used In the calibration 
task. An E^D1 data pair (horizontal and vertical values) oust then be 
locattd as being within one of these quadrangles # From there, two stlmlus 
locations can be obtained, using before and after calibration Informatloni as 
above, and their distance ap^rt found* These distances. on horizontal and 
vertical dimensions are each then divided by. 2 and these products are 
divided by appropriate scaling values to yield ±IA valuei Indicating the 
accuracy of that data point In horlEontal and vertical L*units« 
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Table 1 ^ 
A List of Items to Report 
in Indicating Eye Movement Data Quality 

Charac;teriitlc8 of the signal 
1. Sampling rate 
2* Delay of signal 

a^ Time required for obtaining informatlbn to calculate eye 
pDaltlon 

b. Further delay until eye position algnal is available far 
samp ling 

c* Further delay until sample is obtained 

d* Additional time Vequired for converting the sample to a usable 
. form 

3. MaKltwrn tracking rate of the eye movement monitoring equipment ^ 

4. Noise characteristics of the signal^ 

5. Drift 

Algorltlv&s used in reducing data 

1« Algorltlm for Identifying beginning of a aaccade 
2^ Algorithm for identifying end of a' saccade 

3« Algorithm for Identifying where the eyes are directed during a 
fixation 

a# Nature^ of the calibration task 
b. Nature of the calibration table 
4« Algorithm /for Identifying distur'bances in the eye movement data 

Accuracy of the eye position data 

1* Degree of short'^term repeatability : 

2« Accuracy of the mapping function 

3. Degree of longer-term repeatability 
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Flpira Captions 

Figure 1# Plotting of hypothetical EMM data from a calibration table 
obtained by having a subject look^dlrectly at 20 stimulus points arranged 
rectan^larly In 4 tawm of 5 points each» A highly nonlinear pattern le 
shown to Illustrate the typea of nonllnearlty Chat can occur. The. shaded 
region corresponds to the area shwn Ip Flguriu 2, 

Flpir,e 2. Plotting of shaded region of Figure I, showing baals for 
mapping a data point onto the stlmlus reglpni using the two-dimensional 
linear interpolation approach described in Appendix B« 
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